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1.  Introduction 


As  a  part  of  the  research  project  on  the  force  protection  surveillance  system  (FPSS)  at  the  U.S. 
Army  Research  Laboratory  (ARL),  a  number  of  video  sequences  were  collected  at  the  Adelphi 
Laboratory  Center  (ALC)  of  ARL  to  aid  the  FPSS  development  efforts.  To  achieve  a  generic 
representation  of  typical  FPSS  scenarios,  while  avoiding  specific  details  pertaining  to  ALC 
infrastructures,  all  image  sequences  in  this  dataset  were  taken  from  the  roof  of  a  4-story  building 
and  focused  on  the  main  parking  lot  in  the  direction  of  a  public  road.  No  permanent  buildings  or 
other  significant  infrastructures  in  ALC  were  captured  in  these  sequences.  People  and  vehicles 
were  the  primary  moving  objects  selected  in  this  data  collection  effort.  Due  to  significant 
distances  and  thermal  signal  properties,  no  facial  identity  of  any  human  target  or  license  plate 
number  of  any  vehicle  can  be  recognized  in  these  sequences. 

The  first  85  selected  image  sequences  of  this  data  collection  effort,  which  are  collectively 
referred  to  as  the  “First  ARL  FPSS  Dataset,”  have  been  documented  in  an  ARL  Memorandum 
Report  entitled  “A  Description  on  the  ARL  Force  Protection  Surveillance  System  Data  Set” 
(ARL-MR-600),  and  published  in  October  2004  by  the  same  author.  The  image  sequences  in  the 
first  dataset  were  captured  by  a  number  of  long-wave  infrared  (LWIR)  cameras  and  color  charge- 
coupled  device  (CCD)  cameras  across  different  times  of  the  day  and  different  seasons  of  the 
year.  In  other  words,  the  variability  within  the  first  dataset  was  relatively  high  due  to  significant 
differences  in  the  equipment  used  and  the  background  observed.  In  the  concluding  section  of 
that  Memorandum  Report,  it  was  stated  that  the  usefulness  of  the  dataset  would  be  greatly 
increased  if  the  ground-truth  infonnation  of  the  legitimate  moving  targets  could  be  extracted  and 
used.  Fortunately,  such  ground-truth  information  has  since  been  manually  extracted  from  those 
sequences,  and  these  ground-truth  files  are  now  available  to  the  users  of  that  dataset. 

While  the  First  ARL  FPSS  Dataset  provided  a  variety  of  video  data  that  was  differentiated  by  the 
types  of  sensors,  times  of  day,  seasons  of  the  year,  and  field-of-views  (FOVs),  none  of  the 
recorded  activities  were  purposely  staged  to  represent  one  or  more  suspicious  activities  or 
behaviors  that  would  be  of  interest  to  typical  force  protection  and  security  surveillance 
personnel.  In  other  words,  the  first  dataset  does  not  provide  examples  of  those  rare,  but  possible 
suspicious  scenarios,  that  an  intelligent  force  protection  surveillance  system  should  be  designed 
to  recognize  and  act  upon  in  real-time.  Additionally,  none  of  the  scenarios  were  recorded 
concurrently  by  both  color  and  forward  looking  infrared  (FLIR)  cameras;  therefore,  they  were 
unable  to  support  any  research  on  multi-sensor  detection  and  tracking. 

For  these  reasons,  the  Second  ARL  FPSS  Dataset  was  subsequently  planned  and  collected.  All 
the  video  sequences  in  the  second  dataset  were  collected  using  the  Thermal  Vision  Sentry 
Personnel  Observation  Device  (POD)  manufactured  by  FLIR  Systems,  which  is  shown  in 
figure  1 .  This  Sentry  POD  consists  of  a  LWIR  uncooled  microbolometer  and  a  color  visual 
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camera,  which  are  bore-sighted  and  integrated  in  a  sealed  enclosure.  The  microbolometer  has  a 
focal  plane  array  (FPA)  resolution  of  320  x  240  pixels  and  a  spectral  response  at  the  wavelengths 
of  7.5  to  13  microns.  To  adequately  capture  the  staged  activities  around  the  parking  lot,  the 
wider  FOV  of  this  microbolometer,  24°  by  18°,  was  used  throughout  the  second  data  collection 
effort.  The  color  visual  camera  has  a  quarter-inch  color  CCD,  which  is  capable  of  producing  460 
National  Television  Standards  Committee  (NTSC)  TV  lines  and  zooming  between  2.1°  and  48°. 
The  FOV  of  the  color  camera  was  adjusted  to  about  24°,  so  that  the  resulting  color  and  FLIR 
images  could  be  co-registered  to  an  acceptable  level.  All  scenarios  were  simultaneously 
captured  by  both  cameras  and  stored  in  two  separate  digital  video  recorders  (DVRs). 


Figure  1.  Sentry  POD  manufactured  by  FLIR  Systems. 


2.  Dataset 


In  the  Second  ARL  FPSS  Dataset,  most  of  the  scenarios  were  actively  staged  by  collaborators  to 
represent  a  variety  of  suspicious  activities  and  behaviors  in  the  eyes  of  force  protection  and 
security  surveillance  experts.  These  video  data  were  collected  between  November  2004  and 
January  2005  around  a  parking  lot  at  ALC.  There  are  53  pairs  of  color-FLIR  image  sequences, 
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totaling  71,236  image  frames  and  occupying  3.2  GB  of  disk  space,  in  this  dataset.  A  global 
linear  transformation  method  was  used  to  co-register  each  image  pair  to  an  acceptable  level  and 
scale  the  registered  images  to  a  common  size  of  640  x  480  pixels.  All  images  are  stored  in  JPEG 
format  to  achieve  a  good  trade-off  between  image  qualities  and  file  size.  The  name,  image  type, 
storage  size,  and  number  of  frames  of  each  sequence  are  listed  in  table  1 . 

Table  1.  Listing  of  53  pairs  of  color-FLIR  image  sequences  in  the  Second  ARL  FPSS  Dataset. 


Name  of 

Type  of 

Data  Size  of 

Number  of 

Sequence 

Imagery 

Sequence  (kB) 

Frames 

rf2004 1 120_1 6 1 70 1  fc 

Color 

47536 

990 

rf20041 120  1617016 

LWIR 

27736 

990 

rf20041216_143701fc 

Color 

63400 

990 

rf2004 12 1 6_1 43  70 1  fl 

LWIR 

43576 

990 

rf2004 1 2 1 6_  1 44 1  OOfc 

Color 

63376 

990 

rf2004 121 6 _ 144 1  OOfi 

LWIR 

43576 

990 

rf2004 121 6_ 14423 Ofc 

Color 

63376 

990 

rf2004 1216  1 442306 

LWIR 

43576 

990 

rf2 004 1 2 1 6_ 1 44442 fc 

Color 

63376 

990 

rf2004 1 2 1 6  1 44442fi 

LWIR 

43576 

rf20041216_144715fc 

Color 

63376 

990 

rf2004 121 6  1 447 1 5fi 

LWIR 

43576 

990 

rf20041216_145010fc 

Color 

63376 

990 

rf2004 1 2 1 6_1 45  0 1  Ofi 

LWIR 

43576 

990 

rf20041216_145145fc 

Color 

63376 

990 

rf2004 121 6_1 45 1 45fl 

LWIR 

43576 

990 

rf20041216_145855fc 

Color 

65244 

990 

rf20041216_145855fi 

LWIR 

43576 

990 

rf2004 1216  1 5003 1  fc 

Color 

31448 

490 

rf2004 1 2 1 6  1 5003 1 fi 

LWIR 

21568 

490 

rf2004 1216  1 50427 fc 

Color 

38048 

590 

rf2004 121 6  1 50427fi 

LWIR 

25972 

590 

rf20041216_150612fc 

Color 

12664 

190 

rf2004 121 6  1 5061 2fi 

LWIR 

8364 

190 

rf20041216_150734fc 

Color 

29620 

440 

rf20041216_150734fi 

LWIR 

19368 

440 

rf2004 1216  151 520fc 

Color 

47372 

740 

rf2004 1 2 1 6_1 5 1 520fi 

LWIR 

32572 

740 

rf20041217_132617fc 

Color 

63376 

990 

rf2004 121 7_1 326 1 7fl 

LWIR 

44068 

990 

rf20041217_132750fc 

Color 

63376 

990 

rf2004 121 7  1 327506 

LWIR 

44560 

990 

rf20041217_133143fc 

Color 

63376 

990 

rf2004 121 7_1 33 1 43fl 

LWIR 

43640 

rf2004 1 2 1 7  1 3 345 8fc 

Color 

63376 

rf2004 121 7_1 3345  8fl 

LWIR 

46028 

rf2004 121 7_1 33  7 1 8fc 

Color 

28168 

3 


Name  of 

Type  of 

Data  Size  of 

Number  of 

Sequence 

Imagery 

Sequence  (kB) 

Frames 

rf2004 121 7  1337 1 8fi 

LWIR 

20016 

440 

rf20041217_133858fc 

Color 

31368 

490 

rf2004 121 7_1 33  858fl 

LWIR 

21568 

490 

rf200501 10_172124fc 

Color 

10564 

240 

rf200501 10_172124fi 

LWIR 

6724 

240 

rf200501 10_172252fc 

Color 

8364 

190 

rf200501 10_172252fi 

LWIR 

5324 

190 

rf200501 10_172434fc 

Color 

23704 

590 

rf200501 10  1 724341! 

LWIR 

16532 

590 

rf200501 10  1 72604 fc 

Color 

7900 

190 

rf200501 10172604H 

LWIR 

5080 

190 

rf200501 10_172750fc 

Color 

13648 

310 

rf200501 10_172750fi 

LWIR 

8688 

310 

rf200501 10_172844fc 

Color 

14244 

340 

rf200501 10_172844fi 

LWIR 

9528 

340 

rf200501 10_173000fc 

Color 

8092 

190 

rf200501 10_173000fi 

LWIR 

5324 

190 

rf200501 1  l_160600fc 

Color 

42732 

890 

rf200501 1  l_160600fi 

LWIR 

22808 

890 

rf200501 1 1 _ 1 608 1 5fc 

Color 

47536 

990 

rf20050 1  111  608 1 5fi 

LWIR 

24156 

990 

rf200501 1  l_161040fc 

Color 

47536 

990 

rf20050 1 11  161 040fi 

LWIR 

25200 

990 

rf200501 1  l_161210fc 

Color 

13928 

290 

rf200501 1  l_161210fi 

LWIR 

6968 

290 

rf200501 12_1 1101 9fc 

Color 

42696 

970 

rf200501 12_1 1 101 9fi 

LWIR 

27408 

970 

rf200501 12_1 1 1205fc 

Color 

19368 

440 

rf200501 12_1 1 1205fi 

LWIR 

12468 

440 

rf200501 12_1 1 1350fc 

Color 

43576 

990 

rf200501 12_1 1 1350fi 

LWIR 

28676 

990 

rf200501 12_1 12007fc 

Color 

39176 

890 

rf200501 12_1 12007fi 

LWIR 

28188 

890 

rf200501 12_1 12255fc 

Color 

12768 

290 

rf200501 12_112255fi 

LWIR 

8128 

290 

rf200501 12_1 12338fc 

Color 

13648 

310 

rf200501 12_1 12338fi 

LWIR 

8696 

310 

rf200501 13_09543  lfc 

Color 

47536 

990 

rf200501 13_09543  lfi 

LWIR 

27724 

990 

rf200501 13_095601fc 

Color 

28332 

590 

rf200501 13_095601fi 

LWIR 

16528 

590 

rf200501 13_104347fc 

Color 

39616 

990 

rf200501 13_104347fi 

LWIR 

27736 

990 

rf200501 13_104555fc 

Color 

25612 

640 

rf200501 13_104555fi 

LWIR 

18044 

640 

rf200501 13_104815fc 

Color 

29532 

738 
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Name  of 

Type  of 

Data  Size  of 

Number  of 

Sequence 

Imagery 

Sequence  (kB) 

Frames 

rf20050 1 1 3  1 048 1 5fi 

LWIR 

21636 

738 

rf200501 13_105900fc 

Color 

12008 

300 

rf20050113_105900fi 

LWIR 

8408 

300 

rf200501 13  1 10120fc 

Color 

9604 

240 

rf20050 1 1 3  1 101 20fi 

LWIR 

6724 

240 

rf200501 13  1 10246fc 

Color 

28412 

710 

rf200501 13  1 10246fi 

LWIR 

19892 

710 

rf200501 13_125815fc 

Color 

17608 

440 

rf200501 13_125815fi 

LWIR 

12328 

440 

rf200501 13_130001fc 

Color 

17608 

440 

rf20050 1 1 3_1 3000 1  fl 

LWIR 

12328 

440 

rf200501 13_130320fc 

Color 

5604 

140 

rf200501 13_130320fi 

LWIR 

4048 

140 

rf200501 14_163232fc 

Color 

55456 

990 

rf200501 14_163232fi 

LWIR 

35576 

990 

rf200501 14_163442fc 

Color 

49856 

890 

rf200501 14_163442fi 

LWIR 

32056 

890 

rf200501 14_163600fc 

Color 

49856 

890 

rf200501 14_163600fi 

LWIR 

32056 

890 

rf200501 14_163720fc 

Color 

38652 

690 

rf200501 14_163720fi 

LWIR 

24852 

690 

rf200501 15_120400fc 

Color 

25976 

590 

rf200501 15_120400fi 

LWIR 

21128 

590 

Total 

3,194,404 

71,236 

The  numerical  part  of  the  sequence  name  identities  the  date  and  time  of  the  first  frame  of  the 
sequence.  For  example,  the  first  frame  of  rf20041 120_161701fc  was  recorded  at  16:17:01  h  on 
November  20,  2004.  Because  all  the  sequences  in  this  dataset  were  recorded  at  the  rate  of  10 
frames  per  s,  the  last  frame  of  this  990-frame  sequence  was  recorded  99  s  after  the  first  frame. 
The  last  character  of  the  sequence  name  signifies  the  type  of  sensor  that  produced  the  sequence, 
where  “c”  and  “i”  indicates  the  “color”  and  the  “LWIR”  sensor  type,  respectively.  To  comply 
with  the  naming  scheme  of  the  first  dataset,  the  “rf  ’  at  the  beginning  of  each  sequence  name 
indicates  that  this  sequence  was  collected  from  the  roof  top  of  a  building,  while  the  “f  ’  near  the 
end  of  the  sequence  name  indicates  that  the  sensor  was  made  by  FLIR  Systems. 

The  length  of  each  sequence,  which  varies  between  140  and  990  frames,  was  dictated  by  the 
duration  of  a  given  activity  and  the  1 ,000-frame  limit  of  the  DVR  during  the  retrieval  of  the 
video  data.  Six  long-running  activities  were  split  into  two  consecutive  sequences,  while  the 
longest  scenario  was  split  into  three  consecutive  sequences  due  to  the  limitations  of  our  DVR. 
Obviously,  the  number  of  color  frames  and  the  number  of  FLIR  frames  should  be  the  same  for  a 
given  sequence.  However,  the  file  size  of  a  FLIR  JPEG  image  is  usually  smaller  than  that  of  its 
color  counterpart,  primarily  due  to  reduced  color  dimension  and  scene  complexity.  Figure  2  and 
figure  3  show  a  pair  of  color  LWIR  frames  from  rf200501 10_173000fc  and 
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rf200501 1 0  1 73000fi,  respectively.  These  images  were  linearly  extracted  and  scaled  in  a  way 
that  most  of  the  corresponding  objects  in  the  images  are  nearly  co-registered.  The  co-registration 
is  not  perfect  because  of  the  different  optical  characteristics  of  the  camera  lens.  For  example,  the 
light  post  near  the  upper-left  corner  of  the  color  image  appears  to  be  slanting  leftward,  while 
standing  perfectly  upright  in  the  corresponding  FLIR  image. 


Figure  2.  Color  image  from  the  rf200501 10_173000fc  sequence. 
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Figure  3.  LWIR  image  from  the  rf200501 10_173000fi  sequence. 


Instead  of  producing  calibrated  pixel  intensities  that  are  consistently  tied  to  the  actual  surface 
temperature  of  the  scene  observed,  the  Sentry  POD  constantly  stretches  the  observed  thermal 
profile  across  its  intensity  spectrum  to  maximize  the  contrast  of  the  imagery  outputs.  Due  to  this 
automatic  gain  control  function  of  the  Sentry  POD,  the  resulting  LWIR  signatures  for  a  constant- 
temperature  object,  such  as  a  human,  may  change  based  on  the  background  temperature  of 
different  scenes.  All  the  scenarios  in  the  Second  ARL  FPSS  Dataset  were  collected  during  the 
winter  of  2004,  so  humans  appear  consistently  brighter  than  the  background. 

With  three  exceptions,  every  video  sequence  in  this  dataset  contains  a  suspicious  activity  that 
was  purposely  staged.  In  addition  to  the  staged  activity,  there  were  also  non-staged  activities 
created  by  passerby  persons  and  vehicles  in  35  of  the  50  sequences.  Among  the  three  exceptions 
that  do  not  have  a  staged  activity,  rf2004 12 1 6  1 506 12  and  rf200501 12  1 1 1205  exhibit  some 
non-staged  activities  that  match  the  general  characteristics  of  this  dataset;  therefore,  they  were 
included  in  this  dataset.  In  rf200501 1 5  120400,  there  was  no  observable  activity  captured  in  this 
scenario,  and  it  was  included  solely  to  provide  a  background  reference  to  the  preceding  scenarios 
that  were  taken  under  the  same  FOV.  In  table  2,  we  provide  the  content  description  for  each  and 
every  pair  of  image  sequences  and  describe  not  only  the  staged  activity  involved,  but  also  the 
non-staged  activities  whenever  applicable. 
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Table  2.  Content  descriptions  of  the  sequences  in  the  Second  ARL  FPSS  Dataset. 


Name  of  Sequence 

Description  of  Content 

rf2004 1 120  161 70 1 

2  people  carry  a  box  to  a  car  in  an  empty  parking  lot,  leave  for  a  while,  then  return 
and  put  the  box  into  the  car,  on  a  Saturday. 

rf2004 1216  143701 

3  people  carry  a  box  and  load  it  into  a  van  in  a  filled  parking  lot.  Non-staged: 

4  people  and  4  cars  move  in  the  foreground. 

rf20041216_144100 

3  people  walk  to  a  van  and  open  its  tailgate.  Non-staged:  3  people  and  2  cars  move 
in  the  foreground.  Part  1  of  2. 

rf20041216_144230 

3  people  remove  a  box  from  van  and  carry  it  away.  Non-staged:  1  person  and  2  cars 
move  in  the  foreground.  Part  2  of  2. 

rf20041216_144442 

3  people  carry  a  box  and  leave  it  near  a  hydrant.  Non-staged:  3  people  move  in  the 
foreground. 

rf20041216_144715 

3  people  come  and  remove  a  box  near  a  hydrant.  Non-staged:  2  people  and  6  cars 
move  in  the  foreground. 

rf20041216_145010 

3  people  come  and  loiter  around  vehicles.  Non-staged:  4  people  and  1  car  move  in 
the  foreground.  Part  1  of  2. 

rf20041216_145 145 

3  people  loiter  around  vehicles  and  leave.  Non-staged:  2  people  and  3  cars  move  in 
the  foreground.  Part  2  of  2. 

rf2004 12 1 6_  145  855 

3  people  walk  in  the  wooded  area.  Non-staged:  5  people  and  5  cars  move  in  the 
foreground.  Part  1  of 2. 

rf20041216_150031 

3  people  walk  out  of  the  wooded  area.  Non-staged:  2  people  and  3  cars  move  in  the 
foreground.  Part  2  of  2. 

rf20041216_150427 

A  van  pulls  to  the  curb,  2  people  run  to  it  and  depart  with  the  van.  Non-staged: 

3  people  and  4  cars  move  in  the  foreground. 

rf20041216_150612 

1  person  runs  across  parking  lot,  1  person  and  1  car  also  move  in  the  foreground. 
Non-staged:  This  is  not  a  planned  event. 

rf20041216_150734 

A  van  pulls  to  the  curb,  2  people  get  out  of  it,  and  all  leave  the  scene  later.  Non- 
staged:  1  person  and  1  car  move  in  the  foreground. 

rf2004 12 1 6_1 5 1520 

A  car  pulls  to  the  curb;  another  van  comes  and  stops  next  to  it  for  a  while,  both 
vehicles  leave  afterward.  Non-staged:  2  people  and  1  car  move  in  the  foreground. 

rf20041217_132617 

1  person  comes  and  takes  note  at  the  curb.  Non-staged:  3  people  and  1  truck  move 
in  the  foreground.  Part  1  of  2. 

rf20041217_132750 

1  person  takes  note  at  the  curb  and  leaves.  Non-staged:  3  people  and  2  cars  move  in 
the  foreground.  Part  2  of  2. 

rf20041217_133 143 

1  car  stops  at  curb  for  a  while  and  drives  away.  Non-staged:  2  cars  move  in  the 
foreground.  Part  1  of 2. 

rf20041217_133458 

The  same  car  stops  at  curb  again  and  drives  away.  Non-staged:  2  people  and  3  cars 
move  in  the  foreground.  Part  2  of  2. 

rf20041217_133718 

1  person  leaves  a  bag  near  the  curb  and  walks  away.  Non-staged:  1  car  moves  in  the 
foreground. 

rf2004 121 7_133  858 

1  person  picks  up  a  bag  near  the  curb  and  walks  away.  Non-staged:  2  cars  move  in 
the  foreground. 

rf200501 10  172124 

1  person  leaves  a  bag  in  front  of  trash  box  and  walks  away.  No  other  movement 
observed. 

rf200501 1 0_1 72252 

1  person  picks  up  a  bag  in  front  of  trash  box  and  walks  away.  Non-staged:  1  person 
moves  in  the  foreground. 

rf200501 1 0_1 72434 

1  person  leaves  a  bag  behind  trash  box  and  walks  away.  Non-staged:  3  people  move 
in  the  foreground. 

rf200501 10  172604 

1  person  picks  up  a  bag  behind  trash  box  and  walks  away.  No  other  movement 
observed. 
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Name  of  Sequence 

Description  of  Content 

rf200501 10_1 72750 

1  person  puts  a  bag  under  a  car.  Non-staged:  1  car  moves  in  the  foreground. 

rf200501 10  172844 

1  person  removes  a  bag  from  under  a  car.  No  other  movement  observed. 

rf200501 10  173000 

1  person  walks  past  behind  a  trash  can  with  a  bag,  but  does  not  leave  it  there.  No 
other  movement  observed. 

rf200501 1 1 _ 160600 

1  car  stops  at  trash  can  for  a  while  and  then  leaves.  Non-staged:  2  people  move  in 
the  foreground. 

rf200501 1 1  160815 

1  car  comes  and  parks  in  parking  spot  for  a  while,  and  then  leaves.  No  other 
significant  movement  observed. 

rf200501 1 1  161040 

1  car  parks  in  parking  spot,  the  person  exits  the  car  after  a  while.  Non-staged: 

3  people  move  in  the  foreground.  Part  1  or  2. 

rf200501 1 1  161210 

1  car  parks  in  parking  spot,  the  person  exits  and  walks  away  from  the  car.  Non- 
staged:  2  people  move  in  the  foreground.  Part  2  or  2. 

rf200501 12  11 1019 

1  person  comes  and  takes  notes  near  the  curb,  then  walks  away.  No  other 
movement  observed. 

rf200501 12  11 1205 

Non-staged:  1  person  walks  and  gets  into  a  pickup  truck,  and  then  drives  away.  Not 
a  planned  event.  No  other  movement  observed. 

rf200501 12  11 1350 

1  person  loiters  and  take  notes  near  the  corner  of  parking  lot.  Non-staged:  1  person 
and  2  cars  move  in  the  foreground. 

rf200501 12  112007 

A  car  comes  and  parks  on  a  parking  space,  1  person  leaves  the  car,  loiters  around  the 
curb,  returns  to  car  and  drives  away.  No  other  movement  observed. 

rf200501 12  1 12255 

A  car  comes  and  parks  on  a  parking  space,  1  person  leaves  the  car  in  a  hurry.  No 
other  movement  observed. 

rf20050112_l  12338 

1  person  runs  to  parked  car  and  drives  away  quickly.  No  other  movement  observed. 

rf20050 1 1 3_09543 1 

1  person  comes  and  stretches  extensively  near  the  curb.  No  other  movement 
observed.  Part  1  of 2. 

rf200501 13_095601 

1  person  stretches  extensively  near  the  curb  and  then  leaves.  Non-staged:  1  person 
moves  in  the  foreground.  Part  2  of  2. 

rf200501 13_104347 

1  person  jogs  and  stretches  on  the  sidewalk,  then  leaves.  Non-staged:  1  car  moves 
in  the  foreground. 

rf200501 13_104555 

1  person  climbs  up  the  earth  bank  near  a  pole  from  the  sidewalk  to  the  parking  lot 
and  leaves.  No  other  movement  observed. 

R1200501 13_104815 

1  person  climbs  up  the  earth  bank  near  a  pole  from  the  road  to  the  parking  lot  and 
walks  away.  Non-staged:  1  person  moves  in  the  foreground. 

rf200501 13  105900 

1  person  climbs  up  the  earth  bank  at  a  closer  range  from  the  sidewalk  to  the  parking 
lot  and  walks  across  it.  No  other  movement  observed. 

rf20050113_l  10120 

1  person  walks  up  the  sidewalk.  Non-staged:  1  truck  moves  in  the  foreground. 

rf20050113_l  10246 

1  person  walks  on  sidewalk,  climbs  up  the  earth  bank  at  a  closer  range  and  walks 
across  the  parking  lot.  Non-staged:  1  car  moves  in  the  foreground. 

rf200501 13_125815 

1  car  stops  along  the  earth  bank  beside  a  parking  lot  for  a  while,  no  one  gets  out  of 
the  car,  and  then  the  car  leaves  the  scene.  No  other  movement  observed. 

rf200501 13_130001 

1  car  stops  along  the  earth  bank  beside  a  parking  lot,  a  man  gets  out  of  the  car, 
quickly  climbs  up  the  earth  bank,  hides  between  the  cars  parked  in  the  parking  lot, 
rnshes  back  into  the  waiting  car,  and  then  the  car  leaves  the  scene.  No  other 
movement  observed. 

rf200501 13  130320 

The  same  car  above  drives  along  road  near  earth  bank  without  stopping.  No  other 
movement  observed. 
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Name  of  Sequence 

Description  of  Content 

rf200501 14  163232 

1  person  walks  to  his  car  in  the  parking  lot  and  drives  away.  Non-staged:  2  people 
and  1  car  move  in  the  foreground. 

rf20050114_l  63442 

1  car  comes  and  parks  at  the  end  of  the  parking  lot,  1  person  gets  out  and  walks  on 
the  gravel  road  around  some  trailer  offices.  Non-staged:  2  people  and  2  cars  move 
in  the  foreground.  Part  1  of  3. 

rf200501 14_163600 

1  person  walks  on  the  gravel  road  and  around  the  trailer  offices,  and  then  walks  back 
to  his  car.  Non-staged:  4  people  and  6  cars  move  in  the  foreground.  Part  2  of  3. 

rf200501 14_163720 

The  person  above  drives  his  car  away  from  the  parking  lot.  Non-staged:  2  people 
and  4  cars  move  in  the  foreground.  Part  3  of  3. 

rf200501 15_120400 

The  same  view  as  in  the  scenarios  above  on  a  quiet  Saturday.  Only  one  SUV  parks 
in  the  parking  lot.  No  movement  observed. 

3.  Ground-Truthing  Process 


Carefully  collected,  selected,  and  co-registered  video  data  are  essential  to  the  development  of 
moving  target  indication  algorithms.  However,  it  is  difficult  to  measure  the  performance  of  these 
moving  target  detection  and  tracking  algorithms  without  reliable  ground-truth  information 
associated  with  the  image  sequences.  For  this  reason,  we  have  identified  the  location  and  target 
type  of  all  moving  objects  on  each  and  every  image  frame  in  both  the  First  and  Second  ARL 
FPSS  Datasets.  Given  the  huge  number  of  image  frames  in  these  two  datasets,  we  developed  a 
simple  but  useful  graphical  user  interface  (GUI)  to  facilitate  the  ground-truthing  effort.  The 
resulting  ARL  FPSS  ground-truthing  GUI  is  shown  in  figure  4. 
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Figure  4.  The  graphical  user  interface  for  our  ground-truthing  effort. 


To  use  this  ground-truthing  GUI,  one  begins  by  loading  and  displaying  the  selected  image 
sequence.  This  is  accomplished  by  entering  appropriate  information  into  the  top  three  text¬ 
entering  windows  and  then  hitting  the  “Apply”  button.  The  user  may  then  want  to  play  the 
whole  sequence  a  few  times  to  become  familiarized  with  all  the  movements  existing  in  the 
sequence.  Once  the  user  knows  where  the  moving  objects  are,  he  or  she  may  start  clicking  at  the 
center  of  those  moving  objects  on  any  frame,  while  the  correct  target  type  at  the  lower-left  corner 
of  the  GUI  is  selected.  There  are  five  categories  of  target  type  defined  in  this  GUI,  namely, 
“Person,”  “Vehicle,”  “Animal,”  “Unknown,”  and  “Other,”  which  is  distinctively  represented  by 
a  green  triangle,  a  blue  circle,  an  orange  square,  a  red  cross,  and  a  purple  diamond  on  the  clicked 
locations  on  the  image.  The  “Other”  category  can  be  specified  differently  on  each  frame  as 
necessary,  such  as  lawn-mover,  motorcycle,  ignored-car,  waving-branches,  and  so  forth.  The 
user  does  not,  however,  delineate  a  bounding  box  or  silhouette  for  any  moving  object  on  a  frame. 
There  is  a  trade-off  between  the  extra  effort  spent  and  the  additional  ground-truth  information 
gained.  We  chose  to  focus  our  efforts  with  a  larger  dataset,  than  investing  our  resources  with  the 
tedious  ground-truthing  process. 

The  information  provided  by  the  user  can  be  saved  to  or  retrieved  from  an  American  Standard 
Code  for  Information  Interchange  (ASCII)  ground-truth  file  at  any  time;  hence,  allowing  the  user 
to  stop  and  restart  the  ground-truthing  process  on  a  given  sequence  at  ease.  The  resulting 
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ground-truth  information  can  be  easily  verified  by  loading  an  image  sequence  and  its 
corresponding  ground-truth  file  into  this  ground-truthing  GUI  and  playing  the  sequence  end-to- 
end.  The  markers  will  show  up  on  each  frame  according  to  the  location  and  target  type 
information  provided  by  the  ground-truth  file.  Any  ground-truthing  error  can  be  easily 
recognized  if  the  marker  type  and/or  location  do  not  match  the  moving  objects  on  any  frame. 
When  an  error  is  found,  the  erroneous  marker  can  be  deleted  and  a  correct  marker  can  be  inserted 
with  just  a  few  clicks  of  the  mouse.  Appendix  A  provides  the  ground-truth  information 
pertaining  to  the  first  20  frames  of  the  sequence  of  rf20041216_144100fc.  For  each  frame,  the 
ground-truth  file  provides  the  number  of  moving  targets  in  each  category.  If  the  number  of 
moving  targets  in  a  category  is  not  zero,  then  the  corresponding  X-Y  coordinates  of  each  moving 
target  in  that  category  are  provided. 

As  of  July  2006,  all  45,121  frames  in  the  First  ARL  FPSS  Dataset  and  71,236  frames  in  the 
Second  ARL  FPSS  Dataset  have  been  manually  ground-truthed  and  carefully  verified  using  our 
ground-truthing  GUI.  These  ground- truth  files  are  to  be  distributed,  along  with  the 
corresponding  ARL  FPSS  Datasets,  to  authorized  recipients  of  these  datasets. 


4.  Conclusion 


We  wrote  this  brief  description  to  provide  a  short  but  accurate  overview  of  the  dataset  and 
ground-truth  files  that  we  generated  for  the  FPSS  project  at  ARL.  Although  the  datasets  are  not 
as  extensive  and  complete  as  we  hope  to  one  day  achieve,  they  are  sufficient  to  enable 
preliminary  development  and  testing  efforts  for  moving  target  detection  and  tracking  algorithms. 
Running  the  detection  and  tracking  algorithms  on  a  given  input  sequence  and  comparing  the 
resulting  detections  with  ground-truth  information,  allows  us  to  can  easily  grasp  the  quantitative 
performance  of  these  algorithms  based  on  the  numbers  of  target  hits  and  false  alarms.  Sequences 
with  different  cameras,  targets,  and  background  characteristics  can  expose  the  weaknesses  and 
breakpoints  of  certain  algorithms.  Therefore,  this  dataset  can  be  quite  valuable  for  initial 
development  and  testing  of  detection  and  tracking  algorithms. 

For  further  information  on  the  ARL  FPSS  Datasets,  please  contact  the  author  at:  U.S.  Army 
Research  Laboratory,  AMSRD-ARL-SE-SE  (Attn:  Dr.  Alex  Chan),  2800  Powder  Mill  Road, 
Adelphi,  MD  20783,  TEL  -  301-394-1677,  FAX  -  301-394-5234,  or  e-mail  achan@arl.army.mil. 
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Appendix  A.  Ground-truth  Information 


Ground-truth  infonnation  of  the  first  20  frames  in  rf20041216_144100fc.agt 

Frame :  0 

People:  2  at  x=  103  y=  233  x=  231  y=  217 

Vehicles:  0 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame :  1 

People:  2  at  x=  107  y=  234  x=  233  y=  217 

Vehicles:  0 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame :  2 

People:  2  at  x=  107  y=  234  x=  236  y=  218 

Vehicles:  0 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame :  3 

People:  2  at  x=  106  y=  237  x=  240  y=  218 

Vehicles:  0 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame :  4 

People:  2  at  x=  109  y—  235  x=  242  y=  219 

Vehicles:  0 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame :  5 

People:  2  at  x=  112  y=  235  x=  245  y=  217 

Vehicles:  0 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame :  6 

People:  2  at  x=  114  y=  235  x=  247  y=  217 

Vehicles:  0 

Animals:  0 

Others :  0 

Unknowns:  0 
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Frame :  7 

People:  2  at  x=  115  y=  235  x=  249  y=  217 

Vehicles:  0 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame :  8 

People:  2  at  x=  117  y=  235  x=  253  y=  218 

Vehicles:  0 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame :  9 

People:  2  at  x=  119  y=  235  x=  255  y=  218 

Vehicles:  0 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame:  10 

People:  2  at  x=  120  y=  237  x=  259  y=  219 

Vehicles:  1  at  x=  323  y=  14 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame :  1 1 

People:  2  at  x=  121  y=  240  x=  260  y=  216 

Vehicles:  1  at  x=  330  y=  14 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame:  12 

People:  3  at  x=  123  y=  241  x=  263  y=  216  x=  638  y=  479 

Vehicles:  1  at  x=  333  y=  14 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame:  13 

People:  3  at  x=  127  y=  241  x=  266  y=  214  x=  635  y=  478 

Vehicles:  1  at  x=  344  y=  11 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame:  14 

People:  3  at  x=  127  y=  241  x=  268  y=  218  x=  633  y=  478 

Vehicles:  1  at  x=  346  y=  11 

Animals:  0 

Others :  0 

Unknowns:  0 
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Frame:  15 

People:  3  at  x=  129  y=  242  x=  271  y—  217  x=  630  y=  478 

Vehicles:  1  at  x=  346  y=  11 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame :  1 6 

People:  3  at  x=  131  y=  242  x=  274  y=  217  x=  628  y=  479 

Vehicles:  1  at  x=  370  y=  11 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame:  17 

People:  3  at  x=  133  y=  242  x=  277  y=  216  x=  626  y=  477 

Vehicles:  1  at  x=  374  y=  12 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame :  1 8 

People:  3  at  x=  135  y=  242  x—  279  y=  216  x=  625  y=  476 

Vehicles:  1  at  x=  388  y—  9 

Animals:  0 

Others :  0 

Unknowns:  0 


Frame :  1 9 

People:  3  at  x=  140  y=  242  x=  282  y=  215  x=  623  y=  476 

Vehicles:  1  at  x=  389  y=  11 

Animals:  0 

Others :  0 

Unknowns:  0 

Frame:  20 

People:  3  at  x=  140  y=  242  x=  285  y=  215  x=  620  y=  475 

Vehicles:  1  at  x=  389  y=  11 

Animals:  0 

Others :  0 

Unknowns:  0 
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Abbreviations  and  Acronyms 


ALC 

Adelphi  Laboratory  Center 

ARL 

Army  Research  Laboratory 

ASCII 

American  Standard  Code  for  Information  Interchange 

CCD 

charge-coupled  device 

DVR 

digital  video  recorder 

FLIR 

forward  looking  infrared 

FOV 

field-of-view 

FPA 

focal  plane  array 

FPSS 

force  protection  surveillance  system 

GUI 

graphical  user  interface 

LWIR 

long-wave  infrared 

NTSC 

National  Television  Standards  Committee 

POD 

personnel  observation  device 
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400  Colleran  Road 
APGMD  21005-5009 


No.  of 

Copies  Organization 

US  Army  Aberdeen  Test  Center 
1  HC  ATTN  CSTE  DTC  AT  TC  N 
D  L  Jennings 
400  Colleran  Road 
APGMD  21005-5059 

US  Army  CERDEC  NVESD 
1  HC  ATTN  AMSRD  CER  NV  MS 
R  Driggers 

1  HC  ATTN  AMSRD  CER  NV  ST  J  Hilger 
1  HC  ATTN  AMSRD  CER  NV  ST 
P  Perconti 

10221  Burbeck  Rd  Ste  430 
FT  Belvoir  VA  22060-5806 

US  Army  ERDC 
1  HC  ATTN  CEERD  TR  S 

7701  Telegraph  Rd  Bldg  2592 
Alexandria  VA  22315 

US  Army  Materiel  Sys  Anal  Actvty 
1  HC  ATTN  AMSRD  AMS  SC  G  Kistner 
1  HC  ATTN  AMSRD  AMS  SC  J  Mazz 
1  HC  ATTN  AMSRD  AMS  SC  R  Wheeler 
392  Hopkins  Rd 
APGMD  21005-5071 

US  Army  Natick  RDEC 
Acting  Techl  Dir 

1  HC  ATTN  SBCN  TP  P  Brandler 
Kansas  Street  Bldg  78 
Natick  MA  01760-5056 

US  Army  PM  NV/RSTA 
1  HC  ATTN  SFAE  IEW&S  NV 
10221  Burbeck  Rd 
FT  Belvoir  VA  22060-5806 

Commander 
US  Army  RDECOM 

1  HC  ATTN  AMSRD  AMR  W  C  McCorkle 
5400  Fowler  Rd 

Redstone  Arsenal  AL  35898-5000 
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No.  of 

Copies  Organization 

US  Army  RDECOM  AMRDEC 
1  HC  ATTN  AMSRD  AMR  SG  W  Pittman 
1  HC  ATTN  AMSRD  AMR  SG  IP 
H  F  Anderson 

1  HC  ATTN  AMSRD  AMR  SG  IR  R  Sims 
Bldg  5400 

Redstone  Arsenal  AL  35809 

US  Army  RDECOM  AMRDEC 
1  HC  ATTN  AMSRD  AMR  WS  PL 
W  Davenport 
Bldg  7804 

Redstone  Arsenal  AL  35898 

US  Army  RDECOM  ARDEC 
1  HC  ATTN  AMSRD  AAR  AEP  S  P  Willson 
1  HC  ATTN  AMSRD  AAR  AER  S 
J  Romano 

Radiographic  Laboratory,  B.908 
Picatinney  Arsenal  NJ  07806-5000 

US  Army  RDECOM  TARDEC 
1  HC  ATTN  AMSRD  TAR  R  G  R  Gerhart 
6501  E  Eleven  Rd  MS  263 
Warren  MI  48397-5000 

US  Army  Rsrch  Lab 

1  HC  ATTN  AMSRD  ARL  Cl  OK  TP 
Techl  Lib  T  Landfried 
Bldg  4600 

APGMD  21005-5066 
Director 

US  Army  Rsrch  Lab 

1  HC  ATTN  AMSRD  ARL  RO  EL  L  Dai 
1  HC  ATTN  AMSRD  ARL  RO  M  D  Arney 
1  HC  ATTN  AMSRD  ARL  RO  MI 
R  Zachery 

1  HC  ATTN  AMSRD  ARL  RO  MM 
M  H  Chang 
PO  Box  12211 

Research  Triangle  Park  NC  27709-2211 

US  Army  Soldier  &  Biological  Chem  Ctr 
1  HC  ATTN  AMSSB  RRT  DP  W  Loerop 
Edgewood  Chem  &  Biological  Ctr 
Bldg  E  5554 
APGMD  21010-5424 


No.  of 

Copies  Organization 

1  HC  Commander 

USAISEC 

ATTN  AMSEL  TD  Blau 
Bldg  61801 

FT  Huachuca  AZ  85613-5300 

1  HC  AFRL/SNAA 

ATTN  M  Jarratt 

2241  Avionics  Circle  Area  B, 

Bldg  620 

Wright  Patterson  AFB  OH  45433-7321 

1  HC  CMTCO 

ATTN  MAJ  A  Suzuki 
1030  S  Highway  A1A 
Patrick  AFB  FL  23925-3002 

SITAC 

1  HC  ATTN  H  Stiles 

1  HC  ATTN  K  White 

1  HC  ATTN  R  Downie 

11981  Lee  Jackson  Memorial  Hwy 
Suite  500 

Fairfax  VA  22033-3309 
US  Army  Rsrch  Lab 

1  HC  ATTN  AMSRD  ARL  WM  BF  G  Haas 
APGMD  21005-5067 

US  Army  Rsrch  Lab 

1  HC  ATTN  AMSRD  ARL  WM  BF 

W  Oberle 
Bldg  390 

APGMD  21005-5067 
Director 

US  Army  Rsrch  Lab 
1  HC  ATTN  AMSRD  ARL  RO  EV 
W  D  Bach 
PO  Box  12211 

Research  Triangle  Park  NC  27709 
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No.  of 
Copies 

Organization 

2  HC 

US  Army  Rsrch  Lab 

ATTN  AMSRD  ARL  Cl  OK  T 

2  HC 

Techl  Pub  (2  copies) 

ATTN  AMSRD  ARL  Cl  OK  TL 

1  HC 

Techl  Lib  (2  copies) 

ATTN  AMSRD  ARL  D  J  M  Miller 

1  HC 

ATTN  AMSRD  ARL  SE 

1  HC 

J  Pellegrino 

ATTN  AMSRD  ARL  SE  S  J  Eicke 

30  HC 

ATTN  AMSRD  ARL  SE  SE  A  Chan 

1  HC 

ATTN  AMSRD  ARL  SE  SE 

1  HC 

N  Nasrabadi 

ATTN  AMSRD  ARL  SE  SE 

1  HC 

P  Gillespie 

ATTN  IMNE  ALC  IMS 

Mail  &  Records  Mgmt 

Adelphi  MD  20783-1197 
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